Optimization and Portability of a Fusion OpenACC-based FORTRAN HPC Code from NVIDIA to AMD GPUs

NVIDIA has been the main provider of GPU hardware in HPC systems for over a decade. Most applications that benefit from GPUs have thus been developed and optimized for the NVIDIA software stack. Recent exascale HPC systems are, however, introducing GPUs from other vendors, e.g. with the AMD GPU-based OLCF Frontier system just becoming available. AMD GPUs cannot be directly accessed using the NVIDIA software stack, and require a porting effort by the application developers. This paper provides an overview of our experience porting and optimizing the CGYRO code, a widely-used fusion simulation tool based on FORTRAN with OpenACC-based GPU acceleration. While the porting from the NVIDIA compilers was relatively straightforward using the CRAY compilers on the AMD systems, the performance optimization required more fine-tuning. In the optimization effort, we uncovered code sections that had performed well on NVIDIA GPUs, but were unexpectedly slow on AMD GPUs. After AMD-targeted code optimizations, performance on AMD GPUs has increased to meet our expectations. Modest speed improvements were also seen on NVIDIA GPUs, which was an unexpected benefit of this exercise.Comment: 6 pages, 4 figures, 2 tables, To be published in Proceedings of PEARC2

Turbulent Magnetic Field Amplification from Spiral SASI Modes: Implications for Core-Collapse Supernovae and Proto-Neutron Star Magnetization

We extend our investigation of magnetic field evolution in three-dimensional flows driven by the stationary accretion shock instability (SASI) with a suite of higher-resolution idealized models of the post-bounce core-collapse supernova environment. Our magnetohydrodynamic simulations vary in initial magnetic field strength, rotation rate, and grid resolution. Vigorous SASI-driven turbulence inside the shock amplifies magnetic fields exponentially; but while the amplified fields reduce the kinetic energy of small-scale flows, they do not seem to affect the global shock dynamics. The growth rate and final magnitude of the magnetic energy are very sensitive to grid resolution, and both are underestimated by the simulations. Nevertheless our simulations suggest that neutron star magnetic fields exceeding $10^{14}$ G can result from dynamics driven by the SASI, \emph{even for non-rotating progenitors}.Comment: 28 pages, 17 figures, accepted for publication in the Ap

GenASiS: General Astrophysical Simulation System. I. Refinable Mesh and Nonrelativistic Hydrodynamics

GenASiS (General Astrophysical Simulation System) is a new code being developed initially and primarily, though by no means exclusively, for the simulation of core-collapse supernovae on the world's leading capability supercomputers. This paper---the first in a series---demonstrates a centrally refined coordinate patch suitable for gravitational collapse and documents methods for compressible nonrelativistic hydrodynamics. We benchmark the hydrodynamics capabilities of GenASiS against many standard test problems; the results illustrate the basic competence of our implementation, demonstrate the strengths and limitations of the HLLC relative to the HLL Riemann solver in a number of interesting cases, and provide preliminary indications of the code's ability to scale and to function with cell-by-cell fixed-mesh refinement.Comment: Belated update to version accepted ApJ

GenASiS/GenASiS_Basics: GenASiS_Basics-2.0

This new version announcement accompanying this release is available from Computer Physics Communications, with doi:10.1016/j.cpc.2016.12.019 The full method paper accompanying the original 1.0 version is available at doi:10.1016/j.cpc.2015.06.001 or from arXiv.or

GenASiS/GenASiS_Mathematics: GenASiS_Mathematics-v1.0-beta.1

Initial submission to Computer Physics Communications